feat(rccl): integrate MADEngine workloads into CI [AICOMRCCL-1332] - #9055

Pull request overview

This PR updates RCCL CI to run on a nightly cadence, refactors RCCL test scripts into projects/rccl/ci/scripts/, and adds a new MADEngine-based performance workload (with JSONL datastore + regression detection) for gfx950 on the Ruby cluster.

Changes:

• Switch RCCL scheduled CI from weekly to nightly (06:17 UTC daily).
• Refactor PyTorch/JAX CI workflows to use the new script location under projects/rccl/ci/scripts/.
• Add MADEngine workload runner + reusable workflow to run Llama-3.1-70B training and record/compare throughput over time.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Nice to see MADEngine wired into RCCL CI — this overlaps heavily with what we're
doing for Primus/SGLang multi-node in MAD, so a few notes to keep us on one path.

1. Image distribution to compute nodes

build_rccl_overlay_image() re-implements docker save -> shared staging ->
per-node srun docker load. madengine already does exactly this natively when
the manifest has local_image: true — it saves the image to the MAD_DOCKER_BUILDS
shared dir, workers load the same tar, and it detects a stale image reused under
the same tag via the mad.build_fingerprint label:

https://github.com/ROCm/madengine/blob/ec4de0b58c49f05d89dd33e38cc3e81e0fb3d992/src/madengine/execution/container_runner.py#L2278-L2416

Could you try setting MAD_DOCKER_BUILDS to your shared staging dir and dropping
the custom distribution? If it works on Ruby we'd be on a single mechanism, and
you also get the staleness check for free.

2. Manifest shape + bootstrap interface

The generated manifest puts slurm / env / model_repo / container_mounts at the
top level, but madengine reads deployment_config.{target,slurm,distributed,env_vars}
(see cli/validators.py:154 and primus_backend.py:36). Those top-level blocks are
inert today — the values that actually take effect are the ones duplicated in
--additional-context. Worth moving them into deployment_config so the manifest is
the single source of truth.

Related: nothing sets NCCL_SOCKET_IFNAME / slurm.network_interface. job.sh.j2
only exports it when slurm.network_interface is present, so NCCL bootstrap iface
is left to auto-detection. On Ruby the data-plane ifaces are benic*p1 /31
point-to-point links, so the bootstrap iface should be pinned to fenic0.

We have a static manifest validator that catches both of these (dead config,
missing iface, nodes vs nnodes, leftover placeholders) plus per-cluster env
templates, in https://github.com/ROCm/MAD-private/pull/390 :
.claude/skills/mad-slurm-multinode/scripts/validate_manifest.sh
.claude/skills/mad-slurm-multinode/assets/mad.env/mad.env.thor2-bnxt.template
.claude/skills/mad-slurm-multinode/references/cluster-types.md
Feel free to reuse them.

3. Two things that will break the nightly

• args passes the card name as --model_repo
  ("primus_pyt_megatron_lm_train_llama-3.1-70b_scaleout"), but
  scripts/primus/megatron-lm/run.sh matches base names only
  ("primus_pyt_megatron_lm_train_llama-3.1-70b") with no _scaleout branch and no
  fallback, so $model ends up empty. The model card already carries the correct
  args — dropping the override should fix it.

• Heads-up on a rename: https://github.com/ROCm/MAD-private/pull/388 renames
  those cards from *_scaleout to *_overlay, which will break the hardcoded
  model_repo in WORKLOAD_CONFIGS once it lands. Let's coordinate the timing.

Thanks for pushing this through — 30 fix commits against a cluster madengine had never seen is a
genuinely useful stress test, and nearly everything you had to work around is a real madengine gap
rather than anything wrong on your side.

One request up front, and it is the main point of this review: please don't merge this as-is.
madengine is about to get a lot of active development, so anything that lives here as a patch
against madengine's source will rot — quietly, on a nightly job that no PR check exercises. I'd
like as much of this as possible to move into madengine itself, and only the genuinely CI-specific
parts to stay here.

Before the fix list: run against mad-rccl, not develop

mad-rccl is the MAD branch this CI should be tracking, once
ROCm/MAD#194 lands on it. That branch exists to carry the
RCCL-facing benchmarks; it currently sits at develop's tip (f7e2560, i.e. after Primus v26.5) and
#194 is open against it. It is prepared in parallel with your work and validated on three cluster
archetypes, and a large part of what this PR builds by hand is already in it. Concretely:

• The overlay Dockerfiles — please use these rather than rebuilding the mechanism.
  docker/primus_megatron_train_rccl_overlay.ubuntu.amd.Dockerfile builds RCCL from source over
  rocm/primus:v26.5, takes repo/branch/commit as build args and verifies at build time that
  every librccl on disk matches the built SHA, with optional rdma-core for Broadcom Thor2.
  docker/sglang_disagg_inference_full_overlay.ubuntu.amd.Dockerfile does the same for
  RCCL + MoRI + NIXL/Mooncake in one build. That is the same job as the readelf kpack sniffing,
  the hardcoded ROCm-SDK venv paths and the in-place librccl overwrite at :385-439 — except it
  is a build-time overlay with a verified SHA rather than a post-hoc file swap, and madengine can
  drive it directly through context.docker_build_arg. Swapping a library into a finished image
  and hoping the loader picks it up is the part I would most like to see gone; you already found
  one edge of it when LD_PRELOAD had to be reverted for the LLVM symbol clash.
• The Megatron-LM harness and its metric parsers, scripts/primus_megatron-lm/, writing
  madengine's perf-CSV schema directly. That is what the live-log regex at :681-739 substitutes
  for.
• Real model cards — primus_pyt_megatron_lm_train_llama-3.1-{8b,70b,405b}_overlay in
  models.json. Your workload is primus_pyt_megatron_lm_train_llama-3.1-70b, which is the same
  thing; with a card you would not need to fabricate built_images, built_models and context
  at all.
• Cluster templates and a bring-up skill for CX7/Mellanox-RoCE, AMD-AINIC/Pollara and
  Broadcom-Thor2-RoCE: mad.env plus manifest templates with the transport variables already
  worked out per archetype.

And the one that bears directly on the exit-code discussion below: #194's last commit fixes a
Primus v26.5 field rename that made both metric parsers match nothing. The symptom is exact —
training completes, SLURM reports COMPLETED, every per-node CSV holds only a header, and perf.csv
comes out empty. If you are on a v26.5 base, that is very plausibly the real reason madengine
reports failure on a successful run.

So the concrete ask: point the MAD clone at mad-rccl (pinned to a SHA or tag, per the question
below) instead of public develop, and build the competitor image from the overlay Dockerfile
there instead of assembling one in the CI script. Could we sync before you invest more in the
local versions? It would delete a large part of this diff rather than relocate it.

Moving the fixes into madengine

Two ways to do it, whichever is less work for you:

• hand them to me and I'll land them in madengine, or
• open the PRs against madengine yourself and add @mkuznet1, @i-kosarev and @coketaste as reviewers.

I realise this makes the current task bigger rather than smaller. I still think it is the right
call: this CI integration is a long-lived project, and the alternative is maintaining a patch set
against a moving target.

What I'd like to move upstream. Already written on our side, so close to free — we hit the
same wall on the same cluster, independently of this PR:

• --gpus-per-node emitted unconditionally (job.sh.j2:9) — becomes a manifest flag honoured in
  both the template and the script builder, rather than a line deletion
• the 5 s madengine --version validation timeout (slurm.py:244), too tight for an NFS-backed
  interpreter — raised
• yum info rocm-libs hanging on a GPG prompt (run_orchestrator.py:770) — capped with a
  timeout for every package manager rather than swapped for rpm -qi, since the underlying
  problem is a tty-less prompt rather than yum specifically

Small, not yet written:

• $HOME/.local/bin missing from the sbatch PATH (job.sh.j2:39)
• the shared-FS probe not matching nfs4 (job.sh.j2:144)
• MAD_DOCKER_BUILDS being discoverable only by reading the source

Needs a little design, but still ours:

• running a pre-built image without having to fabricate built_images, built_models and a
  context block for a run that never builds anything
• a headless submit-node mode, so you don't have to invent gpu_renderDs and a HIP version to get
  past the GPU-context guards. Those invented values are the right call given what madengine
  offers today, but they'll be wrong the day the cluster changes and nothing will say so
• a Primus/Megatron perf parser emitting one row per precision, which is what would let the
  live-log scraping go away
• separating "the workload failed" from "metric collection failed", which is what forces the
  exit-code override

What should stay here: the JSONL perf store and the rolling regression gate. madengine has no
perf-history or regression concept in its design and I don't intend to add one — that policy
belongs to the consumer, which is where you already have it.

On the patching helper specifically

patch_madengine_for_cluster() (lines 166-314) rewrites madengine's own
templates/slurm/job.sh.j2, deployment/slurm.py and orchestration/run_orchestrator.py by
string match, after a git clone --depth=1 of madengine's default branch with no pin. Two risks
compound: the clone is unpinned, and every patch fails open — on a miss it logs "already patched
or not found" and continues, so an upstream rename becomes a silent no-op that surfaces days later
as a confusing SLURM error. Given the amount of change coming to madengine, that is close to a
guarantee rather than a risk.

Could you pin the madengine clone to a SHA in the meantime? That alone turns the fail-open
patches from a latent surprise into a deterministic one. The pre-warm at :633-642 also becomes
unnecessary once the probe timeout is raised.

The same goes for the MAD clone (:132-137), and I think it removes the need for the
run.sh grep at :488-499. --model_repo is a MAD-side argument — run.sh parses it at line 33
and the accepted spellings are a MAD contract; madengine never reads it, it just forwards whatever
args says. So checking whether your name is a substring of the script, and falling back through
an alias list if not, is really a hedge against MAD moving under you. Pin MAD to a tag or SHA and
the name is simply known. I'd rather not solve this by having madengine resolve names against
models.json — the manifest carrying the name explicitly is the design we want, not something to
paper over.

On the per-node venv (lines 224-287) — is the Python split actually unavoidable? A venv is
only a thin overlay on the interpreter that created it: it points at the head node's
python3.10 and its compiled extensions are tagged cpython-310, so a 3.9 compute node cannot
use it. That is exactly what you hit, and the block works around it by creating a second venv per
node from that node's own python3 and reinstalling madengine into it.

A conda env doesn't have that property — it ships its own interpreter, so if the env lives on the
shared FS every node sees the same Python and the same site-packages, and there is nothing to
bootstrap per node. That is what we use for multi-node runs and it holds across three different
cluster archetypes:

conda env list | grep -q '^madenv ' || conda create -y -n madenv python=3.12
conda activate madenv
pip install -e ./madengine

with the env on the shared work FS and conda sourced in the job script. If that fits your runner,
the whole per-node install disappears — and with it a pip install on the critical path of every
node — without needing anything from madengine at all. Worth a try before we treat it as an
upstream gap.

Questions before merge — I may be wrong on these

Both of the following come from reading the workflows rather than from running them, and you have
clearly been running this against a real cluster, so treat them as questions.

1. Does the scheduled job actually dispatch? therock-rccl-ci-linux.yml:229 passes
notify_webhook, while therock-rccl-test-madengine.yml declares teams_webhook (lines 23 and
50) and reads inputs.teams_webhook at 156-157. The sibling pytorch/jax jobs at lines 188, 201
and 214 pass notify_webhook because their callees declare it. Reading it cold, that looks like
an undeclared workflow_call input, which would fail validation before the job starts — and since
the job is gated on schedule/workflow_dispatch plus gfx950-dcgpu, no PR check would tell us.
If you've already seen it dispatch, ignore this.

2. Two questions on the HF token. First, is it reaching the job at all?
therock-rccl-test-madengine.yml:147 reads ${{ secrets.HF_TOKEN || '' }}, but I can't find a
secrets: block or secrets: inherit on the call in therock-rccl-ci-linux.yml, which would
make it empty on the scheduled path and fail gated Llama-3.1 access at runtime rather than at
startup.

Second, and more important: can you confirm the token doesn't land anywhere durable? It currently
goes into env_vars (test_madengine.py:486) → into the manifest (:563) → written to disk at
:570, and therock-rccl-test-madengine.yml:178 uploads manifest.json with
retention-days: 30, which is what CodeQL 768 is pointing at (769, the logging one, was fixed by
5631aaae). We had a leak of this kind in madengine and added redaction for MAD_SECRETS_* in
printed commands (fcc4905), but that only covers what madengine itself prints — it won't help
with a manifest written by the CI script and uploaded as an artifact. Passing the token through
the process environment only, and keeping it out of the manifest, would close that off; madengine
picks up MAD_SECRETS_* from the environment, so the manifest doesn't need to carry it.

3. Which MAD are you actually cloning, and is the script copy doing anything? :135 clones
https://github.com/ROCm/MAD.git at --depth=1 with no branch, so that is public MAD's default
branch — develop, unpinned, and it moved as recently as yesterday. Two things there I can't
reconcile:

install_madengine() copies MAD/scripts/primus into the work dir (:147-152), but that path no
longer exists on develop: #187 (Primus v26.5, merged
2026-08-01) deleted scripts/primus/megatron-lm/ in favour of the scripts/Primus submodule —
which is empty after a non-recursive --depth=1 clone — and a convention-based
scripts/primus_train/. The copy is guarded by if scripts_src.is_dir() with no else, so on a
clean checkout it is skipped in silence, and then run_sh.exists() at :491 is false, which makes
the alias fallback underneath it unreachable.

Meanwhile the manifest declares "scripts": "scripts/primus/megatron-lm/run.sh" (:518), and
--work-dir points at a persistent path on the cluster. My guess is that tree was populated once,
by hand or by an older MAD layout, and has been there ever since — which works right up until the
runner is rebuilt.

The fix is the one above: clone mad-rccl at a pinned SHA, where the Megatron-LM harness lives as
scripts/primus_megatron-lm/ and is maintained against v26.5. And whichever ref you end up on,
the copy should fail loudly when its source directory is missing rather than skip — a silent
no-op here is what let this survive a breaking upstream restructure without a single error line.

Worth re-checking

4. Does check_regression() separate precisions? 5b1af729 started writing one record per
precision run, but the filter at test_madengine.py:764-784 looks like it keys on workload, scale
and status only. If so, BF16 and FP8 share one rolling 5-run mean and the −2% gate fires on a
change in precision mix rather than on a regression. Could you double-check whether precision
needs to be in the filter key?

5. The exit-code override, and the empty perf underneath it.
test_madengine.py:1066-1083 forces exit 0 whenever every training run reached its final iteration.
I understand exactly why it is there, but I think the empty perf it compensates for has a specific,
fixable cause: the model entry at :513-524 doesn't declare multiple_results.

That field is how madengine learns where the numbers are. When it is set, madengine passes the
filename into the container as MAD_OUTPUT_CSV, the model script writes that CSV, and madengine
turns each row into a perf.csv row — which is also how you get one row per precision without
scraping anything. When it is absent, madengine falls back to grepping the log for a literal
performance: <value> <metric> line, and Primus never prints one. So performance comes out empty
and the run is marked failed even though training succeeded. The producer for that CSV is the
Megatron-LM report script, which is the other half of question 3 — it isn't on develop any more,
and it is in mad-rccl.

Setting multiple_results and running against mad-rccl should remove the need for both the
override and the live-log regex. Separately, madengine genuinely cannot today distinguish "the
workload failed" from "metrics were empty", and that is on our side to fix; as written the override
will also swallow a real node failure that coexists with a completed training loop. Could you narrow
it to the specific empty-perf condition and log loudly whenever it triggers, until we do?

One note

Thanks for switching to MAD_DOCKER_BUILDS and for completing the NIC config with
slurm.network_interface — both are the right way to use madengine, and the fact that neither is
discoverable without reading the source is on us to fix.

Thanks for taking up the previous round — pinning both clones and moving MAD onto the mad-rccl branch is exactly what we were after. It also means the overlay card and its multiple_results are now available to you directly, instead of having to be reconstructed.

Please model generate_manifest on the manifests we use for manual multinode runs — same MAD branch you pin: https://github.com/ROCm/MAD/tree/mad-rccl/.claude/skills/mad-slurm-multinode/assets/manifests (primus_llama-3.1-70b) — slurm/distributed/env_vars under deployment_config, mounts in context.docker_mounts, multiple_results on the card. Root-level keys are ignored.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

        if current and current.get("iter"):
            current["completed"] = current["iter"] == current["total"]
            runs.append(current)

            image_key: {
                "docker_image": overlay_image,
                "local_image": True,
                "base_docker": overlay_image,
                "build_status": "SKIPPED",

    if workload_config.get("container_mounts"):
        manifest["container_mounts"] = workload_config["container_mounts"]

#!/usr/bin/env python3
"""Run PyTorch c10d NCCL distributed tests against CI-built RCCL.

        "slurm": {
            "partition": slurm_partition,
            "network_interface": network_interface,
            "nodes": nodes,
            "gpus_per_node": gpus_per_node,

Manifest rewrite matches what madengine actually reads (run_orchestrator.py:225-234) — good turnaround.

Acceptance here is multi-node: Llama-3.1-70B on 2 nodes minimum, green, with metrics in perf_entry_super. The description still lists 2N as follow-up; single-node Ruby does not exercise the path this CI exists for.

perf_entry_super is read but unused — metric, regression and datastore still come from the log scraper (inline).